4)

Generate a simulated two-class data set with 100 observations and two features in which there is a visible but non-linear separation between the two classes. Show that in this setting, a support vector machine with a polynomial kernel (with degree greater than 1) or a radial kernel will outperform a support vector classifier on the training data. Which technique performs best on the test data? Make plots and report training and test error rates in order to back up your assertions.

We first create the data set:

We then create our class separation:

We split our observations into train and test sets, run a SVC and a polynomial SVM of degree 2 of over the training data:

Let’s take a look at the training and test error rates between the SVC and the polynomial SVM:

SVC Training Error Rate SVM Training Error Rate
9.803922 19.60784
SVC Test Error Rate SVM Test Error Rate
18.36735 12.2449

Interestingly the linear SVC has a lower error rate over the training data, however the SVM does much better across the test data.

5)

We have seen that we can fit an SVM with a non-linear kernel in order to perform classification using a non-linear decision boundary. We will now see that we can also obtain a non-linear decision boundary by performing logistic regression using non-linear transformations of the features.

a)

Generate a data set with \(n = 500\) and \(p = 2\), such that the observations belong to two classes with a quadratic decision boundary between them.

d)

Apply this model to the training data in order to obtain a predicted class label for each training observation. Plot the observations, colored according to the predicted class labels. The decision boundary should be linear.

We can see the clear linear decision boundary.

e)

Now fit a logistic regression model to the data using non-linear functions of \(X_1\) and \(X_2\) as predictors.

f)

Apply this model to the training data in order to obtain a predicted class label for each training observation. Plot the observations, colored according to the predicted class labels. The decision boundary should be obviously non-linear.

We can see a clear non-linear decision boundary.

h)

Fit a SVM using a non-linear kernel to the data. Obtain a class prediction for each training observation. Plot the observations, colored according to the predicted class labels.

i)

Comment on your results.

We see that we can gain a non-linear decision boundary by using non-linear kernels within our logistic regressions and SVCs.

6)

It is claimed that in the case of data that is just barely linearly separable, a support vector classifier with a small value of cost that misclassifies a couple of training observations may perform better on test data than one with a huge value of cost that does not misclassify any training observations. You will now investigate this claim.

b)

Compute the cross-validation error rates for support vector classifiers with a range of cost values. How many training errors are misclassified for each value of cost considered, and howdoes this relate to the cross-validation errors obtained?

We can use the tune() function to perform the cross validation.

## 
## Parameter tuning of 'svm':
## 
## - sampling method: 10-fold cross validation 
## 
## - best parameters:
##  cost
##   100
## 
## - best performance: 0 
## 
## - Detailed performance results:
##    cost error dispersion
## 1 1e-03  0.44 0.19550504
## 2 1e-02  0.24 0.21705094
## 3 1e-01  0.12 0.09189366
## 4 1e+00  0.02 0.04216370
## 5 1e+01  0.01 0.03162278
## 6 2e+01  0.01 0.03162278
## 7 4e+01  0.02 0.04216370
## 8 1e+02  0.00 0.00000000

333 c)

Generate an appropriate test data set, and compute the test errors corresponding to each of the values of cost considered. Which value of cost leads to the fewest test errors, and how does this compare to the values of cost that yield the fewest training errors and the fewest cross-validation errors?

## # A tibble: 8 x 2
##      cost test_error
##     <dbl>      <dbl>
## 1   0.001         44
## 2   0.01          39
## 3   0.1           43
## 4   1             46
## 5  10             45
## 6  20             46
## 7  40             46
## 8 100             46

d) Discuss

What we see is the bias/variance tradeoff. As the cost is increased, the model becomes more ‘flexible’ and the training error goes down. With the test data, as the flexibility increases we see a decrease in errors until it reaches an inflection point. After this point the model is overfitting and the test error increases.

7)

In this problem, you will use support vector approaches in order to predict whether a given car gets high or low gas mileage based on the Auto data set.

a)

Create a binary variable that takes on a 1 for cars with gas mileage above the median, and a 0 for cars with gas mileage below the median.

b)

Fit a support vector classifier to the data with various values of cost , in order to predict whether a car gets high or low gas mileage. Report the cross-validation errors associated with different values of this parameter. Comment on your results.

## 
## Parameter tuning of 'svm':
## 
## - sampling method: 10-fold cross validation 
## 
## - best parameters:
##  cost
##     1
## 
## - best performance: 0.01025641 
## 
## - Detailed performance results:
##    cost      error dispersion
## 1 1e-02 0.07653846 0.03617137
## 2 1e-01 0.04596154 0.03378238
## 3 1e+00 0.01025641 0.01792836
## 4 1e+01 0.02051282 0.02648194
## 5 1e+02 0.03076923 0.03151981

We see the lowest error when cost = 1.

c)

*Now repeat with radial and polynomial basis kernels and different values of gamma and degree. Comment on your results.

## 
## Parameter tuning of 'svm':
## 
## - sampling method: 10-fold cross validation 
## 
## - best parameters:
##  gamma cost
##   0.01   10
## 
## - best performance: 0.02557692 
## 
## - Detailed performance results:
##    gamma  cost      error dispersion
## 1  1e-02  0.01 0.55115385 0.04366593
## 2  1e-01  0.01 0.21711538 0.09865227
## 3  1e+00  0.01 0.55115385 0.04366593
## 4  1e+01  0.01 0.55115385 0.04366593
## 5  1e+02  0.01 0.55115385 0.04366593
## 6  1e-02  0.10 0.08929487 0.04382379
## 7  1e-01  0.10 0.07903846 0.03874545
## 8  1e+00  0.10 0.55115385 0.04366593
## 9  1e+01  0.10 0.55115385 0.04366593
## 10 1e+02  0.10 0.55115385 0.04366593
## 11 1e-02  1.00 0.07403846 0.03522110
## 12 1e-01  1.00 0.05371795 0.03525162
## 13 1e+00  1.00 0.06384615 0.04375618
## 14 1e+01  1.00 0.51794872 0.05063697
## 15 1e+02  1.00 0.55115385 0.04366593
## 16 1e-02 10.00 0.02557692 0.02093679
## 17 1e-01 10.00 0.03076923 0.03375798
## 18 1e+00 10.00 0.05884615 0.04020934
## 19 1e+01 10.00 0.51794872 0.04917316
## 20 1e+02 10.00 0.55115385 0.04366593
## 
## Parameter tuning of 'svm':
## 
## - sampling method: 10-fold cross validation 
## 
## - best parameters:
##  degree cost
##       2   10
## 
## - best performance: 0.5841667 
## 
## - Detailed performance results:
##    degree  cost     error dispersion
## 1       2  0.01 0.6019231 0.06346118
## 2       3  0.01 0.6019231 0.06346118
## 3       4  0.01 0.6019231 0.06346118
## 4       5  0.01 0.6019231 0.06346118
## 5       2  0.10 0.6019231 0.06346118
## 6       3  0.10 0.6019231 0.06346118
## 7       4  0.10 0.6019231 0.06346118
## 8       5  0.10 0.6019231 0.06346118
## 9       2  1.00 0.6019231 0.06346118
## 10      3  1.00 0.6019231 0.06346118
## 11      4  1.00 0.6019231 0.06346118
## 12      5  1.00 0.6019231 0.06346118
## 13      2 10.00 0.5841667 0.07806609
## 14      3 10.00 0.6019231 0.06346118
## 15      4 10.00 0.6019231 0.06346118
## 16      5 10.00 0.6019231 0.06346118

For a radial kernel, we see a minimisation of ther error when gamma = 0.01 and cost = 10. With the polynomial kernel the lowest error is with degree = 2 and cost = 10.

8)

This problem involves the OJ data set which is part of the ISLR package.

a)

Create a training set containing a random sample of 800 observations, and a test set containing the remaining observations.

b)

*Fit a support vector classifier to the training data using cost = 0.01, with Purchase as the response and the other variables as predictors. Use the summary() function to produce summary statistics, and describe the results obtained.

## 
## Call:
## svm(formula = Purchase ~ ., data = oj_samples$train, kernel = "linear", 
##     cost = 0.01)
## 
## 
## Parameters:
##    SVM-Type:  C-classification 
##  SVM-Kernel:  linear 
##        cost:  0.01 
## 
## Number of Support Vectors:  450
## 
##  ( 226 224 )
## 
## 
## Number of Classes:  2 
## 
## Levels: 
##  CH MM

d)

Use the tune() function to select an optimal cost . Consider values in the range 0.01 to 10.

## 
## Parameter tuning of 'svm':
## 
## - sampling method: 10-fold cross validation 
## 
## - best parameters:
##      cost
##  0.015625
## 
## - best performance: 0.1603557 
## 
## - Detailed performance results:
##           cost     error dispersion
## 1   0.00390625 0.1638577 0.03795702
## 2   0.00781250 0.1626813 0.03729402
## 3   0.01562500 0.1603557 0.03840728
## 4   0.03125000 0.1650205 0.03522022
## 5   0.06250000 0.1638988 0.03695773
## 6   0.12500000 0.1615458 0.03716560
## 7   0.25000000 0.1615595 0.03601742
## 8   0.50000000 0.1603967 0.03777655
## 9   1.00000000 0.1650479 0.03445874
## 10  2.00000000 0.1662244 0.03721419
## 11  4.00000000 0.1627086 0.03446296
## 12  8.00000000 0.1627223 0.03666065
## 13 16.00000000 0.1638851 0.03643699

f)

Repeat parts (b) through (e) using a support vector machine with a radial kernel. Use the default value for gamma.

## 
## Call:
## svm(formula = Purchase ~ ., data = oj_samples$train, kernel = "radial")
## 
## 
## Parameters:
##    SVM-Type:  C-classification 
##  SVM-Kernel:  radial 
##        cost:  1 
## 
## Number of Support Vectors:  390
## 
##  ( 201 189 )
## 
## 
## Number of Classes:  2 
## 
## Levels: 
##  CH MM
## # A tibble: 1 x 1
##   `Train Error Rate`
##                <dbl>
## 1              0.145
## # A tibble: 1 x 1
##   `Test Error Rate`
##               <dbl>
## 1             0.186
## 
## Parameter tuning of 'svm':
## 
## - sampling method: 10-fold cross validation 
## 
## - best parameters:
##  cost
##     1
## 
## - best performance: 0.1673461 
## 
## - Detailed performance results:
##           cost     error dispersion
## 1   0.00390625 0.3764843 0.06228157
## 2   0.00781250 0.3764843 0.06228157
## 3   0.01562500 0.3764843 0.06228157
## 4   0.03125000 0.3366211 0.07569987
## 5   0.06250000 0.1918605 0.04001821
## 6   0.12500000 0.1755267 0.03617482
## 7   0.25000000 0.1755404 0.03986210
## 8   0.50000000 0.1720246 0.04204614
## 9   1.00000000 0.1673461 0.03585901
## 10  2.00000000 0.1720520 0.04109274
## 11  4.00000000 0.1731874 0.04013555
## 12  8.00000000 0.1825581 0.03589686
## 13 16.00000000 0.1837346 0.03185391
## # A tibble: 1 x 1
##   `Train Error Rate`
##                <dbl>
## 1              0.152
## # A tibble: 1 x 1
##   `Test Error Rate`
##               <dbl>
## 1               0.2

g)

Repeat parts (b) through (e) using a support vector machine with a polynomial kernel. Set degree = 2.

## 
## Call:
## svm(formula = Purchase ~ ., data = oj_samples$train, kernel = "polynomial", 
##     degree = 2)
## 
## 
## Parameters:
##    SVM-Type:  C-classification 
##  SVM-Kernel:  polynomial 
##        cost:  1 
##      degree:  2 
##      coef.0:  0 
## 
## Number of Support Vectors:  467
## 
##  ( 238 229 )
## 
## 
## Number of Classes:  2 
## 
## Levels: 
##  CH MM
## # A tibble: 1 x 1
##   `Train Error Rate`
##                <dbl>
## 1              0.174
## # A tibble: 1 x 1
##   `Test Error Rate`
##               <dbl>
## 1             0.214
## 
## Parameter tuning of 'svm':
## 
## - sampling method: 10-fold cross validation 
## 
## - best parameters:
##  cost
##     8
## 
## - best performance: 0.1708208 
## 
## - Detailed performance results:
##           cost     error dispersion
## 1   0.00390625 0.3764843 0.06228157
## 2   0.00781250 0.3776471 0.06301831
## 3   0.01562500 0.3495075 0.07242409
## 4   0.03125000 0.3389740 0.06929994
## 5   0.06250000 0.3179617 0.06929113
## 6   0.12500000 0.3016005 0.07149584
## 7   0.25000000 0.2268673 0.03547776
## 8   0.50000000 0.2012312 0.04937234
## 9   1.00000000 0.1918194 0.04481021
## 10  2.00000000 0.1848290 0.04773197
## 11  4.00000000 0.1801505 0.04354956
## 12  8.00000000 0.1708208 0.04408733
## 13 16.00000000 0.1732011 0.04348912
## # A tibble: 1 x 1
##   `Train Error Rate`
##                <dbl>
## 1              0.137
## # A tibble: 1 x 1
##   `Test Error Rate`
##               <dbl>
## 1             0.219

h)

The radial kernel appears to give the best results on the test data.